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ABSTRACT" 

Anthropological methodology suggests f there may be two 
difficulties with utilizing evaluations over and above the many * 
outlined in several- generations. of evaluation utilization literature. 
The- first - "going natiVe" - arises when the evaluator has 
consciously adopted the value and belief system of those in the 
program he or she is evaluating. The second problem - ethnocentrism - 
arises when the evaluator has been sufficiently blinded by his or her 
own value system that he or she fails to see, and thus take~ into 
ac&ount^ divergent values that characterize audiences or recognize 
how those' values reshape, objectives or goals. Both type's of . problems 
create overt and subtle bias in evaluation reports which in turn 
leads "to skewed judgments, suspect recommendations, loss of evaluator 
and evaluat^m credibility, and legitimation of the failure to 
-utilize evaluation results. Solutions to both problems are developed 
and include-, among,othe& strategies, the audit .(including means for 
establishing an audit trail) , the reflexive' journal, and peer 
debriefing. glt.Author/GK) 
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Abstract 

EIQ EVALUATORS WEAR GRASS* SKIRTS? ' "GOING 
NATIVE" AND ETHNOCENTRISM AS PROBLEMS IN ' 
v UTILIZATION 
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Anthropological methodology suggests there may be 
two difficulties -with utilizing evaluations ov^r and' 
.above the many outlined in Several generations of eval- 
uation utilization literature* The first - "going na- 
tive" arises wh^n the evaluator "has consciously adopted 
the value aifd belief system of those in the program he 
is evaluating . This tendency to be co-opted creates 
-several problems: multiple or competing value systems 
may riot be displayed as clearly or sharply as they 
^^hould be; results may be, skewed toward the favorable r 
r- 3and unfavorable ones may be downplayed of deliberately; 
omitted ^ - . s - : 

The ^second-problem - ethnocentrism - arises when 
the evaluator has^l>een sufficiently blinded by his own 
value 'system t^at' he "fails, to see, and thus" take into 
aqcount, divergent values that characterize audiences . 
or recognize how those values reshape objectives or 
goals. The first instance is a special case of eValu- 
ator -"corruptibility" ; trnei second, naivete or lack of 
evaluator introspection. 

: Both types of problems create overt or subtle bias 

in" evaluation reports which in turn leads to skewed 
judgments .suspect recommendations , loss of evaljuator. 
and jevaluat^on credibility, arid legitimation of the 
failure to utilize evaluation results.' *The usefulness 

.■ of such evaluations for policy decisions or analysis - 
is compromised and"" the likelihood that recommendations 
that will be utilized is" greatly decreased. Neither 
audiences - whos&. vilues are found! to^e not honored - 

' nor policy shapers * who sense they have incomplete or 
erroneous information are moved to act upon reports \ 
which embody either. of these two faults. 

Solutions, to both problems are "developed and include, 
amdng other strategies, the audit (including means for' 
establishing, an iudit trail) , the reflexive journal, and 
peer debriefing. | * ^ 
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As. evaluation has come of age as a practice and .pro- 
fession, eyaluators and' consumers of evaluation\studies 
have l?ecome s increasingly sophisticated in 'confronting the ^_ 
problems facing the field. So, while some writers have 
identified over forty separate evaluation "models" from 
which a practitioner or consumer might choose, other wri-» 
t-ers £ave concerned themselves with barriers to. implementing 
evaluation results and. with reasons why the reports of pro- ' 
fessiona-ls are not more widely utilized. As a result,- the* 

'03 * 
» . * 

"utilization" literature is now nearly as prolific as -the 
•theory and model-building^ literature- of yesterday. 

Policy analyses,; on site interviews and other methods 
have been, employed in the attempt 1k>. discover' why evalua- 
tions are sp^rarely, or at' -least incompletely f utilized. 
Among%the reasons cited can be found several categories v ^ 
for failure. The first'of those categories might be clas- 
sified as- 'utilization problems inherent in evaliHtfrftm ' 
design . In this category of utilization failures' might be 
included the issues of relevance, timeliness and validity 
(Attkisson; Brown and Hargreaves, 1^78); poor data manage- 
ment ^Niegher, 1979, pp. 125-146); and failure "o'f the 

f > S ■> 6 

evaluation tp conform to criteria, for a 'good- evaluation" 
(Guba, 1975). " j • / 

The second category might be classified as the evatttia - 

tion perceptions problem in utilizationV^ TiT this class of * 

utilization failures resides a 'series 'of differing opinions * 

* i " * * * 

about what evaluation can. do, "what it is- supposed to^do* or 

how it relates to either the program which v^Sfcs" designed or,,. 

ERIC . - ■ - . 4 : . • 
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which is currently in operation (and they may not be the 
fc same) • So, for example, diffe-riftg perceptions of the 

evaluation process .and results 11 held by program evalua- , 

v * 
tors and program managers and funders (Attkisson r \Brown* 

and Hargreaves, 1978), "discrepancies between the plans 
1 i 

for the program being evaluated and the actual operation- 

aliz*ation of those, plan's" and* "tKe rapidity of change which 
■ • 

may render evaluation results obsolete before there is time 
to use them 11 (Guba, 1975) may ajl cause evaluations wfrich 
conform to perfectly acceptable .standards of good practice 
to -be utilized less fully than might have, been warranted.. * 

The third cause of failure -to utilize might be traced 

* J % 

* , V 

*to human and political factors . This set of Causes for' 

* _ * 

.failure range from the macro level,' where "poor organiza- 
tional response to evaluation findings" can cause diffi- , 
<culty (AJttkisson, Brovn and Hargjeaves, k978) to the^e ^- ' 

sonar level. In this last subset wotfld be included incon- 

* * • » 

sistent rt>le demands which interfere with the evaluation * 
process, and unwillingness for, one ^or several persons to 
'/sponsor 11 or take responsibility for implementing the eval- 
uation results (Neigher, 1979). Finally, is a third subset of' 
the human and political factors (for example, threat level, 
^ecurity of jobs for personnel, and the like), and the cor- 
ruptibility of evaluators themselves (Guba, 1975, pp. 5^-52)^ 

The methodological^lite'rature in anthropology suggests 
that the evaluator may, be subject t;o two difficulties which^ 
Van result in ignoring > devaluing ar underjxti lining his ^ 
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findings. The first, going native , arises .when the evalua- 
has 'consciously 'adopted the value and belief system of those 
m the program being evaluated , principally those of 'one or 
more of the relevant audiences. The second difficulty, 
• ethnoceritrism , arises when the evaluatar has been sufficiently 
blinded by his own value system that he . f a.il^. to see, *and 
therefore take in account, the. divergent value's which charac-' 
terize his, audience (s) or t,o. recognize how thos'e values "have 
reshaped program goals or objective-sY^ The first, 'instance is 
a special case of evaluator corruptibility' (Guba , l?7£p ;* the 
second, an instance of naivete, inexperience, or lack of . 
evaluator. intr*sp«ction.' This paper wiU examine the impli- J 
cations of t'hos^e- two pp'ssiMlit ies . ' u 

Going Native , 
/ — 1 . w 

Wh-en an anthr,opologis ( has become so Hke - the group he 
is studying that he ceases to consider himself a part of the - 
profession - or ceases to consider/ either his. cultural or pro- 
fessional" sub-group -as his dominant reference group - he is ■ 
said to have "gone native' 1 - (Paul , 1953/ p. 438). 4 He quits 
contributing to the* research ,and begins a /'performance - 
under st and iiig" role (Kolaja, l*S>56 r , p. 161) within* the studied' 
group. Paui, in a discussion -of this problem, named Frank. • 

Cushtfng a"s an example of an anthropologist^hb simply refused 

-. - S " \ 

jto continue publishing the. results $$■ his field studies. 



Identification with the^"natj.ves" , or'\o-op|:ation , as a per'- '." 
sistent problem of Mnguirer- identification, has been a part 
of the "warnings' and advice" given to new participant observers 



for several decades. ' Gold (1969) suggests that going native 
is almost always a re'sult of naivete, and happens as an unfor- 
tunate accident.- In the process of attempting to*^f^.n Ver- . 
stehen, he asserts? "... the field worker may over-identify ' 
with the informant and start td lose his research perspective 
by 'going native*" (p. 56). Moreover, "prolonged direct par- 
ticipation^entails the risk that the researcher will lose 
hiy detached woi>der and fail to discover certain phenomena 
that the relat ively 'uninvolved researcher- would., discoyer" 
(p. 63%54, italics added). Gold has most" assuredly given 
anthropologists ' the' benefit of the doubt, but . it is nit 
certain that jvodern evaluators (save : perh/aps the youngest 
and most inexperienced) would happen* into "going^pa^ive" 
because of over-identif iCation , and one wonders whether the 
benefit of the doubt is' appropriate for them. ' ; 

The problem of going native has_Jbme„._f reshness' for* eval- 
uators. The're are many social act ion "progr-ams -which address 
the pressing needs of virtually "disenfranchised groups of 
citizens, and every egalitarian iftstinct causes one to wish 
them suocess and to want to do. something which will ensure 

that 'success. -Humanitarian urges cau^se irsr-to minimize nega- 

► - 

tive consequences in favor of clearly positive outcomes^ - 
Muiti'ple or competing value systems may not be* d i splay ed-and 
contrasted as clearly or sharply as they should be to have 
the evaluation considered fair; result s,may be skewed toward 



the favorable 'or positive^ and unfavorable effects of, side- 
effects may be downplayed or deliberately omitted. * 

But allowing oneself to be co-opted is likely to result 
in -distorting or overlooking two vital ^areas: (1) the true 
, needs of the "natives", or program audience (s) , for worth- 
while programs, and (-2) the broader social needs "for discrim- 
inating but .fair decisions to "be made.from among a variety 
of potentially worthwhile but resource-scarce programs. -Thus,, 
while the evaluator may feel he is serving a larger social' 
purpose»by attending to 'positive aspects of the program and 
minimizing negative aspects, he may actually be perpetrating' 
a- disservice to both program targets and society at large. 
The boundaries of. his faithlessness to the task at hand--- 
sound evaluation with discriminating choices displayed for 
all groups to see- -are sufficiently wide to cause disrepute 
to settle'- on the evaluation community's a whole. Indeed, 

commented Radin, "True part icipa>dion is simply out of the 

« « * * 

question and romantic participation obscures the situation. . 

'completely. For any "ethnologist to imagine that . anything 

can be gained by 'going native 1 is a deiug^n 'and -a' snare f) 

(1933, p.. Z27 , italics -added) . ' v 

' < * ' V - ' ' 

* . Ethnocentrism 

0 ' ' ' ' " 

Gold*, in discussing problem's- in collecting and analyzing 

field data comments that there is a flip side to the "going 

native 1 ' coin 7 : ) 

' . - ' * l 

"... .with -respect to achieving rappo.rt in a 
fierd relationship, ethnocentrism may be con-' 
'-'sidered the logical opposite' bf, "going native 11 . 
Ethnocentrism occurs whenever *a * field worker 
cannot or will not interact meaningfully with 
an informant M .(1969, p. 37) r 



Vidich confirms enthnocentrism as a legitimate influence in 

the failure to collect and interpret meaningful data, and 

comment/ that is it particularly a» problem when a fieTd worker 

♦(read: evaluator) enters into a setting in his own culture: 

"This naive attitude canng£ be assumed /of 
being, a stranger to one's own culture/ Tn 
working in his own culture* for the simple 
reaspn that the'" respondent canoot accept it 
as plausible . In fact, ""the difficulty of 
securing data may be Increased by the 
1 ethhocentrism 1 • of some respondents who 
'assume that their own experiences are simi- 
lar to those of others." (p. 82) 

Gold and Vidich imply\ that et*hnocentrism works in both 

directions; both on the part of the evaluator and on the part 

' s * 

of the respondents, who may feel, that- the evaluator shares 
their experiences and therefore has a common and' shared set 
of value? and beliefs. i\ r hile neither instance services evalu- 
ation needs ^ ethrfbcentr ism on the part of the evaluator is 

the more crucial, .since it may cause him to miss components of 

v 

cultures or attitudes which are crucial to functioning of 
pi'frgreftis. . ^ • • 

The problem of ethnocentr ism causes particular difficulty 
when the evaluator, must deal Vith pluralistic value contexts. 
Then'one of his principal roles is to discover, explicate .and 
contrast the competing value sets held by various audiences. 
Another important task is to determine the consequences of 
pursuing^ any and all given value structures in order to reflect 
their implications for program direction and operation. 'To * 
t1%6 extent thar he is incapacitated by this ethnocentrism in 
determining structures exist in. a, given program setting, he 



has served the evaluation effortless fully than he might.'- 
. How Serious Are The Problems? " ' • ' 
These warnings from the anthropological literature are 
, t really somewhat more serious than th'ey appear, to be. As part 

\, of the earliest methodological literature in the field,' they •„ 

deal With one of the more serious potential flaws to field- 
• K • work (and by extension., to. evaluation) , bias. Bias. is one* ' 

r 

of the more frequent charges leveled at evaluations, and one 
•of the more subtle of the political factors which militate 
- against using ^valuation results. Bias is an especially 
potent charge when the subject is naturalistic evaluations, 
which rely heavily on the identification, comparison • and 
interpretation of competing value structures in program ' 
sites. The' Rinds of research and evaluation'which the nat- 
- uralist carries outS«sn so strongly on the human as instru- 

ment and his expertise and judgment in discovering va*lues 
and displaying, them in contrasting form that both audiences 
for the evaluation within the program and critics of the 

* * 

; • evaluation (or program) without may deliberately choose to ■ 

ignore reports which they can see conveniently labeled as > 
~ v biased. 

Some* Responses 
How does. one go about avoiding these pitfalls to eval- 
uation? Both going native and ethnocentr-ism^jeate overt or 
subtle bias in evaluation reports which in .turn leads to 
skewed- judgments }N suspect recommendations, los? ^>f evaluator* 
a^d evaluation credibility, and legitimation of the failure 

EKb "" in ■ 



to utilize evaluation-results. The usefulness of such 
evaluations for .polity decisions or analysis is^ compromised 
and the likelihood that recommendations will -be utilized is 

greatly decreased. Neither audiences- -whos^values are found 

- * 

to be disKonored, or not honored at all--nor policy makers 

and ^shapers--who. $<ense they have incomplete § or erroneous 

information- -are moved to act upon reports which embody either 

of these two faults/ 

The quest i6Tr of possible 'responses is' twofold: on the 

one hand, how can an evaluator avoid one or both of these faults 

and prevent their occurrence in himself; on the other hand, how 

can those who receive such reports* be certain they have a rei 

£ort., which is free from $he same biases? The answer to the 

questions involves three strategies - »- the 'reflexive journal, <.'^ 
> / 

peer debriefing, and audit ing- -which help the evaluator 
to develop self-awareness and help others to ."".track" the de- 
velopment of his insights, data and interpretations. These 
strategies., increase the possibility of utilization of evalu- 
ation results . since they mean more publicly inspectable re- 
■ * i • 

suits and invite ^"empar is on with other, equally public inter- 

, « • ■' ^ 

pretations. Each will be discussed in turn. m , 

Strategies for Improving the Changes of Utilization of Evaluation 

Th e Reflexive Journal . Progoff (19-75) has pointed out that 

journals have purposes much larger than merely keeping track 

of -one's day-td-day-r-activities. They can, in fact, provide- 
6 \ v 

major thematlic analysis units for exploring one's "life scripts" 
and for re-writcing and' re -thinking varied aspects of «'those 

.: . •'■ • • ' ' •' . 



•scripts ffhich one may find, dissatisfying , pain-producing or 
unfulf illing. In addition, such a. journal can offer retro- 
spective evidence 'about changes in attitudes, values *ahd* 
•beliefs in the writer (i.e., the evaluator) over time. Like- 
wise, Reinharz (1978) has indicated that reports'of inves- 
tigations typically describe the problem and th^e method; and., 
presumably, if one knows problem and method, One can' then 
tell whether outcomes- and conclusions are trustworthy, rig- 
orous, and/or relevant. But, she counters, research repre- 
sents an integration -not only of problem and method, but also 
of the person who is doing it. To some ' extent , Reinnarz' s 
book (1978)' is a retrospective reflexive journal, as are cer- 
tain writings of Wax (1971), Mead (1972) , and Zigarmi and 
Zig'armi* (197-8) . The effort to understand how the inquirer 
changed as a result of his owt> inquiries , or, as Wax des- 
cribed it j "... the things I learned " (1>9'71, p. 363), is a 

» - 

perennial problem for the reflective observer , and more so 
because of the jiature of some forms of inquiry. 

In 1 the case of naturalistic^ evaluat ion in- particular , 
designs tend to be emergent ;, that i5/vhat happVfis pn any * 
given day depends on what has happened on all the days which 
tame before. 9 It is essential that there be some -means for 
continubusly evolving, summarizing, and projecting, to pro- * 
vide for orderly emergence ' and evolution of the design and 
data collection e'fforts. * * 



. It is also the case in naturalistic evaluations (those 

which do not rely on ijreordinate and fixed 'designs which pre- 

specify each step, in the evaluation process) that one good 

method for establishing at least some aspects of trustworthy- 

* > 
riess (or rigor) is the audit* The reflexive f ie.ldwork" j our- 

nals.. of the- evaiuator provide a rich resource in the audit 

trail. * *. 

To be reflexive means to turn back rn on; thus the re- 

flesive journal turns back on* the inves1;igatbr or evaiuator 

and reflects , him _to himself (and, after the evaluation or 

A 

inquiry, to .others) . Not only is it then clear how the de- 
sign Qf the evaluation is emerging, but it. also becomes 
clear how the evaiuator f s qualities as a "smart" instrument 
(and inquirer) are also* emerging • 

A "relatively complete" reflexive journal probably con- 
tains the following five section^: 

1. V A log of evolving perceptions . T t his log begins with 
the Written perceptions of the evaiuator prior to his" entrance 
on site; and additional statements are written at regular in- 
tervals-. The separate renderings may be kept in escrow by 
someone^ elsfe^ -usually an auditor or peer debrief er- -who is 
not directly connected with the program t or project evaluation. 
At the end of .the p'roject (or during its duration if it is of 
^sufficient time la*pse) , these perceptual statements may be 
compared to one another and/or to aspects of the design and/or 
analysis to check whether learning is occuring, whether origi- 
nal perceptions and beliefs pexseverate, whether later findings 



arip 'clouded by these perceptions, and whether or ndt the 
evaluator. has "learned 1 ; anything "new" or changed himself. 

'• ?• s A log of day-to-day procedures .. 'The purpose of this' 
log is to indicate in diary or chronographic form exactly 
what -was accomplished every day. The most important use of 
this component of the journal* will be an the evaluation audit 
process. ' " *. 

'3: A log of methodological decision p.oi-nts . This par-, 
ticuiar log of" the reflexive journal is entered on an 'ad hoc 
basis as necessary, and should record all major methodological 
decisions / such as explicating new design steps, decisions on 
instrumentation, finalizatio^ of an analysis; category set or 
the like, and such decisions should be entered Together with / 

reasons or rationale f or! %he action, taken or decision made. ' 

* *i 

The auditor is the chief audience for this log. 

A log of day-to-day personal, introspections . Here 
one lays out in diary foyn^one's own thoughts and feelings, 
including stresses one is almost bound to undergo (Zigartoi 
and Zigarmi, 1978) and frustrations one encounters, and how 

v. * > 

those feelings and situatiorfs are perceived to be changing. 
It is important to have some insight into one's self and to 
work' on geheraring that insight. The chief use to which 
this *lag can be pu£ is to test for bias in the evaluatbr 
and to' relate, decisions about "design and procedures to it ^ * 
later. 
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*^ A log of developing insights- and hypotheses , ' The 

emergent design wrll depend almost exclusively on how the 

evaluator takes advantage of what he has already learned.- 

The purpose of" this section is to keep readily available 

,an up-to-date summary of" where 'one is with respect to 

-> « 
knowledge of\the situation and working hypotheses about it. 

Those working hypotheses which have been discarded ox "out- 

* 1 * 

grown" ought to be relegated to historical files on the 
project* °and retained as- part of the audit trail. There 
are several, uses for this .log, including the guiding* of the 
inquiry, shaping of ttie emergent design, providing the basis 
4or 'subsequent' data collection and* analyzing actiyities, 

and' for post hoc auditing procedures , • * * 

• * * * * 

One of the reasons so. few such f "journals 11 are kept is 
that the keeping of one requires enormous* discipline for an 
'evaluator, Especially when he knows that his entries will be 
. . open *to inspection, for instance, Jby debrief ers and perhaps 
^ later by auditors. The logs provide an ongoing record of 
the plans, operations, decisions y travails and other ntenta} 
-processes and activities of the evaluator. 



Peer De-b^iefing . This particular .strategy for maintaining 
one's, sense of reality- counteracts botji the process of' "going 
lf\ : - native" and ethnocentrism, but it also contributes a third 

. ~ v - benefit to the ^ongoing evaluation: it allows for working * 

* - ' * * * 

hypotheses to be tested by a peer or colleague who- may or v 

. ^ $ 

may not be involved with the evaluation firsthand. Peer 
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de-briefing can take place either wit|h a person .or persons 

'* '*„'*' * : - ' 

outside and removed from' the- evaluation ' effort , or with members 

of" a team of evaluators who are operating in a split;-h~alf or 
teaming frode, collecting independent sets of data, then re- 
viewing "those data for discovery and verification 'of 'meanings * ' 
and findings. In either instance, the purpose of such de- 
briefings \s -to keep the eyaluator--and fiis data collection 
and interpretation prQcedures--"on track," -usually 'by reflect- 
ing on a priori written statements of the evaluator, by. re- 
viewing ttfe reflexive journal <and by careful questioning 'of 
the data which are reported to the' debri^fer. 

There are three 'reasons why an evaluator might w*ant to * 
engage a peer de-brie^etr. . The 'first is relief from the pressures 
of long' and hard days' of trying to "fit" or coping with long 
and complicated, intefitiews which must the^n be- transcribed, 

to 1 ' , 

fleshed out and completed late at night and aloh-e, or the v 
strains, of observation, especially / when one is apt ce'rtain " 
what one is 'observing, make for the necessity of having some- 
one -simply , to talfc *to (better yet-, to act, as a formal debrief er) 
(Zigarmi and'Zigarmi, 1978). 

The second is- to recover from what Wax emails "immersion," , 

• • • ft * 

by which is meant"^. .a joint process (*6f becoming a member-of 

a society or culture of livirfg people) ... involving -numerous 

accommodations and adjustments , by both the fieldwdrker and-* the 

people who- 'accept 1 - him" (1971, p. 43, italics the author's 

own). She describes forms of immersion (which occasionally 

~j '. V : 
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lead- to "going native" or otherwise losing* perspective on the 

» 

research) : * • , 

% > \ "For example, a fieldworker may become so fas- 
cinated by the new, exciting, and significant 
things he is learning*, th^t he may .spend months 
passionately^ and persistently thinking of nothing 
else. Simul.tanepu-sly , he will find himself 
becoming personally or socially involved in * 
the community, not only because of his devel- 
oping relatioiis}fip£ with acquaintances, employees, 
and friends, but because, to some degree at 
^least, he is now really beginning to lose" 
touch with his own people and with the world 
outside . . 

N - There is another and deeper kind of im- 

l mersion which may occur after a fieldworker 
« , has truly become involved with the 'living 

people' in the society he is studying. Indeed x 
he maybe unaware that he is 'immersed' until 
he^ is given the opportunity to leave the field 
. for a pleasant vacation and finds that he does 
, ..not want to go. Sometimes, his new^ and hferd- , 
« .won social ties and gelations, may mean, or seem 

to mean, more to him than his ties with his 
own people." (-1971, pp. 43-*44) . - ' 



Thus, even reasonable p*eoplre may disagree on what* the 
problem is, but the,* need for getting £way from* the site, and* 
for having someone .(a peer) debrief the inquirer on hiS re- 
search is imperative. * . . 

A third reason to utilize peer 'debriefing is to prpvide 
for an external check. ' Douglas reports that, when operating 
in a team situation (mo.re than one researcher or eval.uator) , 
"...teajn members do not merely provide support. They also 
provide a' vital check on" each other . FieTd researchers have 
always bemoaned the almost total lack of internal checks and 
of retesting their studies. The few instances of retesting 
that have been done .♦. h^ve shown that they heed it badly. 
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Retests tend to show that each participant observer has' gotten 
at different parts of given different interpretations of things". 
(197b, p. 217, italics .ours) . .. % 

The Audit and AudiJ Trail . Perhaps the best method for insuring 
that evaluators (and indeed, aU field-orieftted, naturalistic in- 
quirers) keep themselves- -and a/e kept- -honest , is the establish- 
ment of an audit' trail. Not only does such a "track" prevent* 
the evaluator 'from becoming so enmeshej^in his own values that 
he fails to discover and portray multiple and competing falue 
systems in the v context, it also allows^ others to discover when 
he^ra^ become so enmeshed in the site Lid the program that he 
ha 6 s "gone 7ta^ve fr or remained Ethnocentric and betrayed his 
. Q real 'purposes. 

» Th^audit is a major technique modelled on the fiscal 

audit which can be used both for dependability (analogous to 

' - ■ ■ . v 

y reliability) and conf irmability (analogous to objectivity) 

s tyirpo^es (Ridings, XL980; Gtfba, 198T; Halpern, 1981). That, the 

techniques will suffice for these purpbses is well illustrated 

be a comment made by Cronbach and Suppes in describing' a 

feature that^distinguishes disciplined inquiry ^rom other forms: 

ri «\ . th^^eport of a disciplined* inquiry has 
• a texturerjiat displays tne raw materials en- 
tering the* a^gi^ment and the logical processes 
by which they w^f^i compressed and rearranged 
to make the conclusion credible . n (1969, p. 16) 

The audit is a means for carrying out this kind of public 

examination. It is b'ased on the metaphor^f the fiscal auditor, 

who, when called in to examine the books of a client corporation 

or business, has two major tasks tha.t he is expected t.o accomplish: 

ERIC ^ .-' ' " : X8 



1. To examine the processes by which the local accounts 
are kept, not so much to assure that there 'has been "no fraud' 
(although the detection of fraud /is \ne of the auditor's functions) 

V 

than to .assure that the books represent a fair statement of the 
company's fiscal position. . The auditor is particularly concerned 
^that there has been no "creative account ing ff , as it- is called in 
the trade, for example, to make the company look more attractive 
to* stock purchasers or to possible buyers. The auditor's major 
taske is that he be able to certify that the processes used by 
th6 local accountants fall within the bounds of acceptable pro- 
fessional practice. ^ 

2. To examine the products of the local accounting pro- 
cesses, to ascertain that -every entry in the books can be 'authen- 
ticated either by direct documentation or by solicited confirming 
statements, „ that the "bottom line" is correct, and that interpre- 
tations made of the accounts in any fiscal statement based on them 
are appropriate. , 

In similar fashion, the inquiry. or evaluation auditor has 

two" tasks: to review the inquiry processes to-be certain that 

* m 
they fall within the bounds, of acceptable professional pract^cjy^ 

(norms) , and tc> review the inquiry products to "be certain that 

they can be substantiated from ttie data collected (a step, by 

the way, wh^ch-is almost never applied systematically 'in conven- 
er S 

tional research). The former~task is equivalent tQ^e&tablishing 
the dependability of the\inquiry and the latter its confirmability 

y 
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C° f both data arid, conclusions)". The issue is not whether • 

r , 
the eyaluator has carried out the processes or reached the . 

conclusions in the same way the auditor would have .done, 

• but whether the evaluator has carried out the processes 

» 

in a reasonable manner. >fhus,' incidentally, replication is not 
called, for as a criterion, but rationality is.. • - . 

The actions of the auditor diffdr depending on whether 'he 
is doing a dependability audit or a conf irmability audit, although 
of course he may do bo$h. In the role of a dependability auditor, 
he will: examine all of the documentation from the point of view 
of its acceptability within the norms of good naturalisticpro- 
fessional practice, and ; certi fy that. the inquiry has been adequately 
and fairly carried out and attest to that fact with a formal state- 
ment. In the role of a conf irmability 'auditor , he will: examine 

• all analyzed data; and compare, a sample of analyzed data with 
original data items such as interview notes or documents to 
satisfy himself that: the data items have been Reasonable unit- 
ized , and categorized , that individual data items have 6een reason- 
able assigned to appropriate taxons or categories; that conclusions 
ire documented in terms of .the category system ; and that conclusions 

• are triangulated with respect to multiple data sources. In addi- 
tion, he will certify that the inquiry products are properly fo#hded 
on the data and reasonably interpreted from them, and attest "to 
that fact with a formal statement. 

p If auditing is to be done .properly, the auditor should be in- 

^ vqlved form 'the beginning to be certain ^at' a proper audit tra^l* 
is being left. The auditor may wi*sh toWork with the investigator* 



throughout the study, although of .course the major'work of the 
auditor* comes post facto* 

L ^ j * 

w - • Summary ^ 

We have tried to indicate -that there are several types jof t 
problems which creep 'into evaluation* reporting which affect the ' 
report f s potential for Utilization. Those problems are suggested 
by the anthropology methodological literature in anthropology 
gating back nearly 50 years, and revolve aWut either the.evalua- 
tor's "going native"-- or consciously or unconsiously adopting 
the value structure of his respondents to the exj^nt £hat he ceases 
to relate %o his professional peer group as a professional --^and 



the inverse of this situation flthnocentr ism failing to recog- 

nize, portray and honor the mult iple. value perspectives he, might 
find in a context because his. own value orientations**may have ~ 
blinded him to other perspectives. t * - • 

, Either of these two forms .of bia^s is* likely to result in* fail- 

l v . * . • • 1 •• :* 

ure to utilize the t results of evaluation; in alienation of* program 

targets;, who realize their value positions are not: honored : in 

the inability of such reports to be employed, in policy analyses; 

and in a loss of evaluator and^valuatioiTxrediljility . 

• We have suggested three strategies whjrch help^ detect aruK 

cqunteract bias in^evaluation reporting .% The . : first , the refleacive 

journal, allows the evaluator, to charters j)wn growth and to mark 

Where he 4 is not- "learnings as program audiences are teaching^hini, 

* ♦ - . '* ' 

or where' he may be unconscious prejudices which are not originally 

~known to him. The second, peer d^£ief-ing , R allbwS. / colleague toll 

provid£ this check, both on the evaluator r s f progress -as a "'smart 11 

instrument, and to verify that the study ^ is proceeding in an appro- 



priate collection and- analytic mode; Tlte/t-hi^, the evaluation 

* . • , \ v * 

audit, provides for a methodological • and analytiCaccount ing by 

* « • * -'*.'., ' . ' •«* 

an outside party to. the evaluation, in order fb insure that proper 

procedures, both with Tespect to "data collection and with respect * 

to data analysis, have been carried out. In. addition, the audit 

provides a means £or suggesting proper., '.'audit trails", qr account's 

of the research and decision points, which wilt 1 extend the mean's • 

for peer reviewers to certify that: the research -has proceeded 

along standard and accepted cenpn's of.-ggod pr^cfi^e* 
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